feat(benchmark): Crossover concurrency: thread pool + rate-limit guard + thread-safe sidecar (byte-identical)#23
Merged
Conversation
…ntical) A full real crossover runs ~15h sequentially; the answering/embedding calls are I/O-bound, so running N cells in flight cuts wall-clock to a fraction. build_dataset is refactored into two phases: Phase 1 computes each cell's outcome (optionally on a ThreadPoolExecutor, each worker using a thread-local answering model so the mutable last_usage never races); Phase 2 assembles the dataset in deterministic grid order. The result is BYTE-IDENTICAL to sequential for any concurrency -- proven on a 2x3 grid (concurrency 1 vs 6) and multi-seed (1 vs 8). util/ratelimit.py adds a thread-safe token-bucket RateLimiter acquired once per cell so the pool self-throttles under a hosted API's RPM cap; it only delays, never changes a call's result. concurrency/rate_limiter thread through build_dataset / run_seeds / build_dataset_multiseed (default concurrency=1 -> unchanged behavior).
The crossover sidecar is now written from worker threads, so _on_cell serialises its whole write+flush+print under a lock -- interleaved writes would corrupt the JSONL that --resume reads back. The demo command gains --concurrency N (cells in flight) and --rpm RATE (token-bucket cap), threaded into build_dataset / build_dataset_multiseed / the augment run_seeds path. Proven end-to-end: a --concurrency 6 CLI run produces a byte-identical crossover_dataset.json and an uncorrupted sidecar (every line valid JSON, every cell exactly once).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A full real crossover runs ~15h sequentially; the answering/embedding calls are I/O-bound, so running N cells in flight cuts wall-clock to roughly 1/N. Delivered in the three pieces requested, and proven byte-identical to sequential on a small grid. Full suite green (477 passed, 6 skipped).
1. Concurrency pool (thread pool, not async)
build_datasetis refactored into two phases:ThreadPoolExecutor(max_workers=concurrency). Each worker uses a thread-local answering model so the mutablelast_usagenever races across cells. Cell computation is a pure function of(seed, N, arm, scenario)— the corpus is deterministic — so outcomes don't depend on completion order.Because assembly is order-deterministic, the dataset is byte-identical for any concurrency. Threaded through
run_seeds/build_dataset_multiseed; defaultconcurrency=1leaves behavior unchanged. (The batched builder already parallelises server-side via the Batch API, so it's untouched.)2. Rate-limit guard
util/ratelimit.py— a thread-safe token-bucketRateLimiter(rpm)acquired once per cell, so the pool self-throttles under a hosted API's requests-per-minute cap. It only delays calls, never changes what one returns, so results stay byte-identical.3. Thread-safe sidecar writes
The durable
.partial.jsonlsidecar is now written from worker threads, so_on_cellserialises its whole write+flush+print under a lock — interleaved writes would corrupt the JSONL that--resumereads back.CLI
demogains--concurrency N(cells in flight) and--rpm RATE(token-bucket cap), threaded into the single-seed, multi-seed, and--augmentpaths.Proof (the deliverable)
tests/test_concurrency.py:concurrency=1vsconcurrency=6→ identicaljson.dumps(sort_keys=True); same for multi-seed (1 vs 8).--concurrency 6demo run produces a byte-identicalcrossover_dataset.jsonand an uncorrupted sidecar (every line valid JSON, every cell exactly once — no interleaving).Independent of PR #20 (roster Step 1) and PR #22 (Holm/CI-clamp); branched from
main.